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ABSTRACT 

Rollover  incidents  while  driving  their  military 
vehicles  has  resulted  in,  soldiers  incurring  injuries  or 
even  losing  their  lives.  A  recent  report  identified  that 
one  cause  of  vehicle  rollovers  is  the  driver's  inability 
to  assess  rollover  threat,  such  as  a  cliff,  soft  ground, 
water,  or  culvert  on  the  passenger  side  of  the 
vehicle.  The  vehicle’s  width  hinders  the  driver's  field 
of  view.  To  reduce  the  number  of  military  vehicles 
rolling  over,  a  road  edge  detection  and  driver 
warning  system  is  being  developed  to  warn  the  driver 
of  potential  rollover  threats  and  keeps  the  driver  from 
veering  off  the  side  of  the  road.  This  system  utilizes 
a  unique,  ultra-fast,  image-processing  algorithm 
based  on  the  neurobiology  of  insect  vision, 
specifically  the  vision  of  a  fly.  The  system  consists  of 
a  Long-Wavelength  Infrared  (LW1R)  camera  and 
monochrome  video  camera  system,  a  long-range 
laser  scanner,  a  processing  module  in  which  a 
biomimetic  image  processor  detects  road  edges  in 
real-time,  and  a  Driver’s  Vision  Enhancer  (DVE) 
which  displays  the  road  image,  detected  boundaries 
and  road-side  terrain  steepness  in  real-time  for  the 
driver. 

1.  INTRODUCTION 

A  crew  of  two  soldiers  is  in  a  High-Mobility 
Multipurpose  Wheeled  Vehicle  (HMMWV)  driving  at 
night  without  the  headlights  turned  on,  on  an 
unpaved,  unmarked  road.  Unexpectedly,  a  camel 
crosses  the  road  and  the  driver  quickly  swerves  to 
the  side  of  the  road  to  avoid  hitting  the  camel. 
Unbeknown  to  the  driver  and  the  passenger,  there 
was  a  twelve  foot  ditch  on  the  passenger’s  side  of 
the  vehicle.  The  HWWVW  rolled  over  and  ejected 
the  soldiers  out  of  their  seats,  since  neither  of  them 
was  wearing  seatbelts.  Neither  of  the  soldiers  was 
seriously  injured,  but  they  did  suffer  a  broken  arm 
and  a  sprained  neqlj£\This.  accident  is  only  a 
theoretical  example  of  Wnat  soldiers  can  experience 
while  driving  military  vehicles.  A  recent  report 
identified  that  one  cause  of  rollover  is  the  driver's 
inability  to  assess  rollover  threat,  such  as  a  cliff,  soft 
ground,  water,  or  culvert  on  the  passenger  side  of 


the  vehicle.  The  vehicle’s  width  hinders  the  driver’s 
field  of  view.  Each  year,  Army  personnel  and 
soldiers  are  involved  in  military  vehicle  accidents, 
many  of  which  could  be  avoided.  There  are  several 
ways  to  decrease  military  vehicle  rollovers  and 
associated  accident  fatalities;  one  way  is  a  road 
edge  detection  and  driver  warning  system. 
Engineers  and  research  scientists  are  currently 
developing  a  road  edge  detection  and  driver  warning 
system  to  warn  the  driver  of  potential  rollover  threats 
and  to  keep  the  driver  from  veering  off  the  side  of  the 
road. 

2.  ROAD  EDGE  DETECTION  SYSTEM 


Figure  1 :  General  Architecture  of  the  Road 
Edge  Detection  System 

Figure  1  shows  the  conceptual  general 
architecture  of  the  system.  This  diagram  shows  the 
video  and  LWIR  cameras  which  provide  images  for 
daytime,  nighttime,  smoke,  fog,  snow,  and  rain 
operation.  The  laser  scanner  measures  the  terrain 
steepness.  The  video  images  are  read  in  by  a 
processing  unit.  The  algorithms  embedded  in  the 
processing  unit  include  fly  vision-based,  edge  fusion, 
road  edge  identification  and  roll-over  threat 
determination.  The  fly  vision-based  algorithm  is 
applied  to  detect  all  edges  present  in  the  video  and 
LWIR  images.  The  edge  fusion  algorithm  combines 
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the  detected  edges  into  a  single  set  of  edges.  It  is 
part  of  a  processing  unit  that  fuses  images  from  the 
cameras,  extracts  edges  from  the  images,  detects 
road  edges  and  identifies  the  road  boundary  on  the 
passenger  side  of  the  vehicle,  and  integrates  the 
terrain  steepness  information  with  the  road  boundary 
information  to  assess  rollover  threat  and  warn  the 
driver.  The  road  edge  identification  algorithm 
identifies  among  all  the  detected  edges  those 
pertaining  to  the  road  boundary.  The  rollover  threat 
detection  algorithm  detects  the  presence  of  negative 
roadside  terrain  on  the  passenger  side  of  the  road, 
and  warns  the  driver  acoustically  and  visually  using 
alarm  signals. 


Monochrome 
Video  Camera 
(clear  daytime  view) 


LWIR  Camera 
(nighttime  view,  smoke, 
fog,  snow,  rain) 

Planar  Laser  Scanner 
up  to  80  m  detection  range 
j  ■  up  to  90"  scanning  angle 
•  0.5*Resolution 

■  up  to  75Hz  scanning  frequency 


Figure  2:  Video  and  LWIR  Cameras  potential 
Mounting  Positions. 


Road  Side  Terrain  (Cliff) 


Lane  Marker 


Physical  Edge 


Camera  System 
Fleld-of*Vlew 


Laser  Scanner 
Fleld-of-Vlew 


Figure  3:  Potential  Scanning  Range  of  the 
Camera  System. 


Figure  2  shows  an  example  of  how  the 
monochrome  video  and  LWIR  cameras  and  laser 
scanner  will  be  mounted.  The  different  shading  in 
Figure  3  indicates  the  wider  field-of-view  that  the 
camera  system  has  versus  the  laser  scanner.  The 
camera  system  provides  road  boundary  detection 
and  laser  scanner  provides  range  and  roadside 
terrain  steepness  information. 

2.1  Video  Camera 

Video  images  have  been  used  to  detect  and 
track  road  boundaries.  The  images  obtained  from 
the  video  provide  the  best  results  when  there  is 
daylight,  good  weather  conditions  and  on  roads  with 
lane  markers.  Video  images  are  difficult  to  use 
during  nighttime,  rain,  snow,  fog,  smoke  and  on 
unmarked  terrain.  To  improve  viewing  the  scenery 
during  nighttime,  or  in  the  presence  of  smoke,  fog, 
snow,  and  rain,  and  on  all  terrain  conditions,  both 
video  and  LWIR  cameras  are  needed. 

LWIR  camera  is  not  affected  by  bright  sunlight  or 
headlights,  but  it  is  sensitive  to  temperature.  There 
are  two  types  of  infrared  imagery  technologies, 
cooled  and  uncooied  infrared  detectors.  The  cooled 
infrared  cameras  deliver  higher  resolution  images, 
since  the  cameras  often  use  Barium  Strontium 
Titanate  ferroelectric,  but  they  are  bulky  and 
expensive,  each  unit  costs  more  than  $25,000,  which 
is  a  hefty  price  tag  in  today’s  budget  minded  Army. 
The  cooled  infrared  has  an  integrated  thermo¬ 
electric  cooler  (TEC),  which  makes  the  unit  heavy 
and  oversized.  Furthermore,  it  also  consumes  more 
power.  On  the  other  hand,  the  uncooled  infrared 
camera  is  more  affordable  and  compact  since  each 
unit  costs  less  than  $10,000  and  uses  a  micro¬ 
bolometer  that  is  sensitive  to  radiation  in  the  7-14 
micron  band  for  night  time  use. 

The  image  display  from  a  LWIR  camera  enables 
the  driver  to  distinguish  the  different  types  of 
materials  used  on  the  roads,  such  as  asphalt,  dirt,  or 
gravel,  from  the  embankment,  such  as  grass,  rock  or 
cliff.  This  distinction  provides  better  contrast,  thus 
making  detection  of  the  road  boundary  and 
embankment  easier  during  nighttime  driving  and  in 
the  presence  of  the  different  weather  conditions. 
However,  utilizing  both  the  monochrome  and  LWIR 
cameras  does  not  supply  sufficient  images  for 
detecting  terrain  depression  as  accurately,  since  the 
cameras  do  not  provide  depth  perception.  To 
overcome  this  hurdle,  a  laser  scanner  is  also 
needed. 
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2.2  Laser  Scanner 

The  laser  scanner  provides  depth  or  steepness 
perception  since  the  laser  scanner  can  scan  any 
point  in  space  to  indicate  the  present  of  a  cliff  or  an 
obstacle.  An  increase  in  the  slope  of  the  scanning 
denotes  a  terrain  depression  such  as  a  cliff  that  is  on 
the  passenger’s  side,  while  a  decrease  in  the  slope 
will  denote  a  positive  terrain  such  as  a  big  rock  or  an 
obstacle  that  is  on  the  passenger  side  of  the  vehicle. 

The  laser  scanner  that  the  researchers  and 
engineers  are  considering  has  a  light  beam  that  is 
invisible  to  the  naked  eye  and  should  not  interfere 
with  the  LWIR  imaging.  The  laser  light  beam  should 
operate  in  near  infrared  or  less  than  1  pm  while  the 
LWIR  is  sensitive  only  to  long  wavelengths,  greater 
than  7.0pm.  The  laser  light  needs  to  be 
inconspicuous  and  should  not  signal  the  vehicle’s 
presence  to  the  enemy.  The  laser  scanner  needs  to 
have  at  least  a  resolution  of  10mm,  a  90  degree 
angle  between  the  incident  beam  and  the  reflecting 
material  for  proper  measurement,  and  at  least  a  75m 
detection  range.  A  75m  range  is  needed  because  a 
short  distance  detection  range  is  not  beneficial  to  the 
driver.  With  longer  distance  range,  the  system  can 
detect  steepness  of  the  terrain  far  enough  ahead  of 
the  vehicle  to  warn  the  driver  and  allow  the  driver 
enough  time  to  react  to  avoid  the  obstacle  or  the 
rollover  threat. 

3.  PROCESSING  SYSTEM 

After  the  monochrome  and  LWIR  cameras 
capture  the  images,  algorithms  are  needed  to 
process  the  images.  These  algorithms  are  the  fly 
eye-based  algorithm  for  road  edge  detection,  an 
algorithm  for  fusing  visible  and  LWIR  edge 
information,  a  road  boundary  identification  algorithm, 
and  a  roll-over  threat  detection  algorithm. 

3.1  Road  Edge  Detection  Algorithm 

The  road  edge  detection  algorithm  extracts  the 
edges  in  the  images.  Typically,  images  from  the 
video  require  intensive  computational  power  and 
time.  This  lag  causes  a  delay  of  the  information 
displayed  to  the  user  so  it  is  not  in  real-time.  To 
address  this,  an  innovative  method  to  process  the 
video  images  has  been  developed,  which  requires 
less  computational  power  and  provides  real-time 
information  to  the  user.  This  technique  applies  the  fly 
eye-based  vision.  This  concept  of  the  fly  eye-based 
vision,  or  biomimetric  edge  vision,  is  based  on  the 
neurobiology  of  insect  vision,  since  insects  rely  on 
edge  detection  to  avoid  obstacles.  The  way  an  insect 


uses  its  vision  significantly  reduces  the  intensity  of 
visual  signal  processing  in  the  insect’s  brain. 

An  example  of  using  the  fly  eye-based  edge 
detection  algorithm  is  shown  in  Figure  4a  and  4b. 
Figure  4a  shows  an  image  of  the  road  and  Figure  4b 
shows  the  same  images  using  the  fly  eye-based 
detection  algorithm.  From  looking  at  Figure  4b,  one 
notices  that  the  algorithm  captures  the  relevant 
contour  pixel  of  the  image,  which  are  the  lane  maker, 
cliff  edge  and  road  boundary.  The  double  edge  lines 
on  the  passenger  side  correspond  to  the  lane  marker 
and  the  actual  road  edge  where  the  cliff  begins. 

Since  an  insect  only  sees  the  edges,  software 
leveraging  this  type  of  cognitive  vision  response 
system  increases  the  robustness  and  computational 
speed  of  the  system.  The  fly-eye-based,  edge- 
detection  algorithm  is  thousands  of  times  faster  than 
the  traditional,  gradient-based  algorithm  because 
only  relevant  edge  data  is  utilized.  This  process  also 
requires  less  storage  space  because  only  the 
relevant  edge  information  is  captured.  The  other 
information  is  not  necessary. 


(a)  (b) 

Figure  4:  (a)  Image  of  the  Road  and  (b)  Image 
Using  Fly-eye  Based  Detection  Algorithm  . 

The  monochrome  video  and  LWIR  images 
obtained  from  the  ultra-fast,  fly-eye-based  edge- 
detection  algorithm  need  to  be  fused  to  form  a 
single,  robust  estimate  of  the  detected  edges.  This 
process  is  used  to  increase  the  probability  of 
detection  of  the  road  edge  and  to  decrease  the 
probability  of  a  false  alarm. 

3.2  Fusion  Algorithm 

The  video  camera  and  the  LWIR  camera  will  be 
placed  and  oriented  such  that  both  the  images  share 
roughly  the  same  field-of-view,  so  that  the  images 
captured  by  both  cameras  can  be  superimposed. 
Thus,  the  image  from  each  camera,  once  processed 
through  the  edge  detector,  will  provide  a  set  of 
detected  edges  representing  the  edge  information. 
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The  fusion  process  fuses  the  edges  detected  in 
each  of  the  individual  images  captured  by  the  video 
and  LWIR,  into  a  single,  enhanced  set  of  edges.  This 
method  is  modular  in  the  sense  that  one  detector, 
either  the  video  or  LWIR  cameras,  can  be  removed 
for  a  given  platform  or  application  without  changing 
the  fundamental  algorithm  operation.  This  allows  the 
framework  to  be  applied  to  a  wide  range  of  systems 
of  various  practicality  and  payload. 

The  fusion  algorithm  uses  a  weighted  sum  of  the 
pixel  intensities  of  detected  edges  in  the  visible  and 
LWIR  images.  Thus,  this  method  integrates  the  edge 
information  into  a  single-edge  image.  Equation  1 
shows  how  the  weighted  sum  is  calculated. 

Ifused  (X,y)“  (X  l visible  (X>Y)  "^(l-  C(.)  IlWIrI^iY)  (1) 

In  Equation  1,  lvisibie  (x,y)  and  ILwir(*,Y)  are  the 
intensities  of  the  edge  pixels  irr  the  visible  and  the 
LWIR  images  at  position  (x,y),  and  a  is  the  weight  of 
the  fusion  (0  <  a<1).  To  vary  a,  the  contribution 
from  either  the  LWIR  camera  or  the  video  camera 
can  be  increased  or  decreased.  Figure  5  shows  and 
example  of  the  edge  fusion,  where  a=0.5.  In  Figure 
5,  the  picture  on  the  left  is  of  the  road  taken  at  night 
using  a  monochrome  video  camera,  while  the  picture 
on  the  right  is  with  a  LWIR  camera.  The  bottom 
picture  is  a  superimposed  image  of  the  two  pictures 
using  the  fusion  algorithm. 


3.3  Road  Boundary  Identification  Algorithm 

The  road  boundary  identification  algorithm 
identifies  edges  that  are  part  of  the  road  boundary 
and  reconstructs  the  road  boundary  with  the  selected 
edges  since  not  all  the  edges  are  associated  with  the 
road  boundary.  The  algorithm  applies  a  model-based 
technique  to  fit  consecutive  segments  of  the  road 
boundary  with  a  curve  fragment  by  fitting  the  road 
edge  segment  with  a  short  2D  vector.  This  algorithm 
is  similar  to  the  one  proposed  by  Wilson  (Wilson, 
1999). 

This  algorithm  works  by  automatically  searching 
for  the  origin  of  the  road  boundary  in  the  lower  right 
quadrant  of  the  edges  image  by  looking  for  the  origin 
of  the  bottommost,  rightmost  straight  line.  Then  a 
short  vector  of  selected  length  or  fitting  window  is 
drawn  from  the  origin.  The  algorithm  then  counts  the 
number  of  detected  edges  belonging  to  the  vector  by 
rotating  the  vector  about  the  origin  point  within  a 
region  defined  by  the  original  reference  to  find  the 
best  fit.  The  best  fit  corresponds  to  the  position  for 
which  the  fitting  vector  has  the  most  edge  points  and 
represents  the  detected  fragment  for  the  road 
boundary.  The  tip  of  the  vector  becomes  the  new 
origin.  The  procedure  is  repeated  to  detect  the  full 
road  boundary  fragment  by  fragment. 

3.4  Rollover  Threat  Detection  Algorithm 

Once  the  road  boundary  is  detected,  the 
presence  of  a  negative  terrain  on  the  roadside,  and 
whether  the  vehicle  is  on  course  to  cross  the 
boundary  on  the  passenger  side  and  rollover  are 
also  considered.  In  addition,  the  time  to  road  edge 
crossing  is  computed.  To  estimate  the  road  edge 
crossing,  the  threat  detection  algorithm  uses  the 
vehicle’s  heading,  current  speed  of  the  vehicle,  and 
the  distance  to  the  point  of  rollover.  When  a  terrain 
depression  is  detected  on  the  passenger  side  or 
when  the  road  edge  crossing  is  below  a  certain 
threshold,  which  is  related  to  the  driver  reaction  time, 
an#  acoustic  signal  and  a  visual  signal  will 
simultaneously  warn  the  driver  of  the  danger.  The 
acoustic  signal  could  be  something  similar  to  the 
stall-horn  on  small  airplanes  or  a  beeping  sound,  and 
the  visual  signal  could  be  a  red  flashing  LED  on  the 
GUI  screen  to  indicate  rollover  point.  The  threshold 
will  be  set  according  to  the  driver’s  reaction  time, 
which  will  allow  the  driver  to  steer  the  vehicle  back 
on  track.  The  output  of  the  processing  system  will  be 
displayed  on  a  screen  inside  the  vehicle  using  a  GUI. 


Figure  5:  Edges  Fusion  Example 


3.5  Driver’s  Vision  Enhancer  (DVE) 

The  information  obtained  from  the  LWIR  and 
monochrome  video  camera  system  is  combined  with 
the  digital  signal  processing  unit  to  be  displayed  on 
the  Driver-Vehicle  Interface  (DVI).  The  Driver- 
Vehicle  Interface  is  similar  to  a  Graphical  User 
Interface  (GUI).  The  DVI  allows  the  driver  to  interact 
with  and  control  the  road  edge  detection  system.  The 
DVI  will  be  used  to  display  the  current  image  of  the 
terrain,  such  as  the  road  ahead  as  seen  from  the 
monochrome  and  LWIR  cameras,  the  detected  road 
edges,  and  the  roadside  terrain  steepness 
information.  The  displayed  image  will  be  a  composite 
image  made  of  the  images  captured  by  the  video 
and  LWIR  cameras.  The  driver  will  be  able  to 
initialize  the  edge  detection  and  identification 
procedure  by  selecting  a  point  on  the  screen.  This 
point  will  be  used  as  the  starting  point  for  the 
algorithm.  Other  features  that  will  be  considered  for 
the  display  are  GPS,  compass,  and  dead  reckoning. 
Figure  6  shows  an  example  of  a  potential  DVI 
display  for  the  driver. 
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driver  of  potential  rollover  threats.  This  detection 
system  utilizes  a  unique,  ultra-fast,  image-processing 
algorithm  based  on  the  neurobiology  of  insect  vision, 
specifically  the  vision  of  a  fly.  The  system  consists  of 
a  Long-Wavelength  Infrared  (LWIR)  camera  and 
monochrome  video  camera  system,  a  long-range 
laser  scanner,  a  processing  module  in  which  a 
biomimetic  image  processor  detects  edges  in  real¬ 
time,  and  a  Driver-Vehicle  Interface  (DVI)  which 
displays  the  road  image,  detected  boundaries  and 
road-side  terrain  steepness  in  real-time  for  the 
driver. 

The  rapid,  road-edge  detection  based  on 
biomimetic  image  processing  has  the  potential  to  be 
implemented  in  the  Future  Combat  Systems'  (FCS) 
Automated  Navigation  System  (ANS).  The  road  edge 
detection  system  can  enhance  the  capability  of  the 
FCS  by  navigating  mobile,  autonomous  ground 
robots,  detecting  potential  rollover  threats,  and 
preventing  the  autonomous  vehicle  from  rolling  over. 
Furthermore,  this  system  can  be  embedded  in  FCS 
Manned  Ground  Vehicle  (MGV)  to  reduce  rollover 
accidents,  thus  minimizing  the  number  of  injuries 
and  ultimately  sav&soldiers’  lives. 
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Figure  6:  Conceptual  Design  for  the  DVI  Display 

4.  CONCLUSION 

The  development  of  the  road  edge  detection  and 
driver  warning  system  will  decrease  the  number  of 
rollover  incidents  since  the  system  will  notify  the 
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